lazy snapshot sending by TheSafo · Pull Request #50120 · DataDog/datadog-agent

TheSafo · 2026-04-29T17:23:38Z

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

TheSafo · 2026-04-29T17:24:00Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

jsaf/flatten-json-encoding-values #50349 : 2 dependent PRs (#50402 , #50530 )
extedned json nesting #50347
flat-log-encoding #50321
fix delta encoding #50133
ttl in config #50121
lazy snapshot sending #50120 👈 (View in Graphite)
tag manager addition #50049
don't send deletes, tighten TTL #50047
utf8 #49898 : 2 other dependent PRs (#49961 , #50036 )
fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety #49810
better json #49672
local proto plus servic #49628
feat(logs): periodically remove unused entry from dict #49494
feat(logs): max template size limit + json_as_raw raw bypass #49552
fix(logs): send delete on tag eviction #49493
fix(logs): rebuild libpatterns.so targeting glibc 2.30 (was 2.34) #49446
feat(logs): send JSON via dict-encoded sets of keys #49404
feat(logs): add dual-send shadow gRPC pipeline #49327
feat(logs): wire stateful encoding into log agent pipeline #49220
feat(logs): add pattern extraction to gRPC stateful sender #49219
feat(logs): add pattern clustering, tag management, and JSON processor #49218
feat(logs): add generic eviction framework for pattern management #49217
feat(logs): add Rust FFI tokenizer bridge with FlatBuffers #49216
feat(logs): add stateful encoding module plumbing and token types #49215
[Logs Stateful Encoding] transport layer initial implementation #48879
logs-stateful-main

This stack of pull requests is managed by Graphite. Learn more about stacking.

agent-platform-auto-pr · 2026-04-29T17:32:43Z

Go Package Import Differences

Baseline: 228994b
Comparison: b305a9c

binary	os	arch	change
agent	linux	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	linux	arm64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	windows	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	darwin	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	darwin	arm64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
agent	aix	ppc64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
iot-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
iot-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
heroku-agent	linux	amd64	+11, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust/flatbuffers/patterns +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/google/flatbuffers/go
cluster-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
cluster-agent-cloudfoundry	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
dogstatsd	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
dogstatsd	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	windows	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	darwin	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
process-agent	darwin	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
heroku-process-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
security-agent	linux	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	linux	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
security-agent	windows	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	linux	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	linux	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	windows	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	darwin	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
system-probe	darwin	arm64	+10, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/DataDog/datadog-agent/pkg/trace/log
otel-agent	linux	amd64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
otel-agent	linux	arm64	+9, -0 +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering +github.com/DataDog/datadog-agent/pkg/logs/patterns/clustering/merging +github.com/DataDog/datadog-agent/pkg/logs/patterns/eviction +github.com/DataDog/datadog-agent/pkg/logs/patterns/processor +github.com/DataDog/datadog-agent/pkg/logs/patterns/tags +github.com/DataDog/datadog-agent/pkg/logs/patterns/token +github.com/DataDog/datadog-agent/pkg/logs/patterns/tokenizer/rust +github.com/DataDog/datadog-agent/pkg/logs/sender/grpc +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb
privateactionrunner	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	windows	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	darwin	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers
privateactionrunner	darwin	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/proto/pbgo/statefulpb +github.com/planetscale/vtprotobuf/protohelpers

TheSafo mentioned this pull request Apr 29, 2026

don't send deletes, tighten TTL #50047

Draft

TheSafo mentioned this pull request Apr 29, 2026

tag manager addition #50049

Draft

This was referenced Apr 29, 2026

ttl in config #50121

Draft

fix delta encoding #50133

Draft

flat-log-encoding #50321

Draft

extedned json nesting #50347

Draft

jsaf/flatten-json-encoding-values #50349

Draft

json presence #50402

Draft

This was referenced May 8, 2026

inflight fixes #50530

Draft

jsaf/updated-inflight-eviction #50557

Draft

disable uuid optionally #50637

Draft

lazy snapshot sending

b305a9c

TheSafo force-pushed the jsaf/lazy-inflight-recording branch from a914270 to b305a9c Compare May 15, 2026 17:25

TheSafo force-pushed the jsaf/tag-manager-addition branch from 91f9f86 to 8b3aa13 Compare May 15, 2026 17:25

This was referenced May 18, 2026

cpu mem #50975

Draft

override origin version #50982

Draft

additional headers #51041

Draft

dual strategy reverse #51139

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lazy snapshot sending#50120

lazy snapshot sending#50120
TheSafo wants to merge 1 commit into
jsaf/tag-manager-additionfrom
jsaf/lazy-inflight-recording

TheSafo commented Apr 29, 2026

Uh oh!

TheSafo commented Apr 29, 2026 •

edited

Loading

Uh oh!

agent-platform-auto-pr Bot commented Apr 29, 2026 •

edited by dd-octo-sts Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

TheSafo commented Apr 29, 2026

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

TheSafo commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agent-platform-auto-pr Bot commented Apr 29, 2026 • edited by dd-octo-sts Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Go Package Import Differences

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

TheSafo commented Apr 29, 2026 •

edited

Loading

agent-platform-auto-pr Bot commented Apr 29, 2026 •

edited by dd-octo-sts Bot

Loading